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ABSTRACT 

This paper introduces some recent changes by the current 
administration that are increasing the visibility of program evaluation in 
Washington and then describes some concerns raised by this new attention. The 
paper also suggests ways that evaluators can assist federal decision makers 
in using evaluation results wisely. Over the past decade, Congress and 
several Administrations have put in place a structure for increasing the 
federal focus on accountability for government performance. In accord with 
these changes, the Office of Management and Budget (OMB) introduced a more 
formal assessment process and a tool, the Program Assessment Rating Tool 
(PART), to apply to 20% of programs in preparing the fiscal year 2004 budget. 
The PART program fits with OMB 1 s agency-by-agency budget reviews, but it is 
not well-suited to addressing cross-cutting issues or to looking at broad 
program areas in which several programs address a common goal. PART can be 
used to promote discussions of what is and is not known about program 
performance and then to stimulate evaluations designed to fill the gaps. To 
assist in the integration of evaluation and performance information into 
budget and policy discussions, the paper suggests evaluators could: (1) make 

the evaluation more relevant; (2) help prepare agency staff for budget 
discussions; (3) develop tools to improve the budget and performance 
integration process; and (4) improve the quality of state and local data. 
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“Linking Program Performance to the Budget: Can Agencies Meet the 
President’s Challenge?” Stephanie Shipman, U.S. General Accounting Office . 1 
Paper presented at the annual meeting of the American Evaluation Association, 
November 9, 2002, Arlington, VA. 

In this paper I aim to introduce those of you who are not yet familiar with them to recent 
Administration initiatives that are increasing the visibility of program evaluation in 
Washington, then describe some of my concerns about this new attention, and finally, 
suggest ways that evaluators can assist federal decision makers in using evaluation 
results wisely. I am, of course, presenting my own opinions and not representing GAO’s 
views. 

Over the past decade, Congress and several Administrations have put in place a structure 
for increasing the federal focus on accountability for government performance. A 
central part of that structure is the Government Performance and Results Act of 1993 
(GPRA) that required strategic and performance plans and annual performance 
reporting, and also required links between performance plans and budgets. While 
implementing GPRA has expanded the supply of information on program performance 
and results, many in Congress and the Executive Branch have been frustrated by the 
slow progress of improving government performance and accountability. In August 
2001, the President released a Management Agenda to advance the process and address 
“the government’s most glaring problems” with five government-wide initiatives that 
together aim to help government achieve better results through increasing efficiency, 
effectiveness, and accountability. 2 The five initiatives are: strategic management of 
human capital; competitive sourcing to create a market-based government; improved 
financial management; expanded electronic government; and budget and performance 
integration. These reforms track very closely to the federal statutory framework of 
management reforms enacted during the 1990s. 3 

To encourage accountability, the Administration is tracking how well departments and 
agencies are executing these management initiatives with an Executive Branch 
Management Scorecard. Last winter, the Office of Management and Budget (OMB) rated 
agencies’ progress in implementing these reforms, and published agencies’ 2001 baseline 
scores - as red, yellow and green traffic lights. The budget and performance integration 
initiative, in particular, raises the visibility of agencies’ evaluation activities. OMB’s 
criteria for obtaining a “green fight” on budget and performance integration are: 

1) an integrated planning/evaluation and budget staff work with program managers 
to create an integrated plan/budget and evaluate its implementation, 

2) a streamlined plan/budget sets forth outcome goals, targets and resources 
requested in the context of past performance, 

3) budget accounts, staff, and activities are aligned to support program targets, 

4) full budgetary cost is charged to mission accounts and activities, and 



' The views expressed are the author’s, and do not represent U.S. General Accounting Office policy. 

2 The President’s Management Agenda, Fiscal Year 2002. Executive Office of the President, Office of 
Management and Budget. N.d., n.p. http://www.whitehouse.gov/omh/hudintegration/pma index.html 

3 Managing for Results: The Statutory Framework for Performance-Based Management and Accountability 
(GAO/GGD/AIMD-98-52, Jan. 28, 1998). Available at htt p://www. gao. gov 
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5) the agency has documented program effectiveness and can show how outputs and 
policies affect desired outcomes and how program results inform budget 
decisions. 

These standards are high; no department received a green light on budget and 
performance integration last January, and only three received a yellow light (EPA, 
Transportation, and the Small Business Administration). 

Federal program evaluations were further spotlighted when OMB incorporated 
assessments of the effectiveness of a selected set of programs in the President’s Fiscal 
Year 2003 budget proposal. Those first round assessments were a little rough and not 
always clearly sourced, but this spring, OMB introduced a more formal assessment 
process and a tool, the PART - Program Assessment Rating Tool - to apply to 20 percent 
of programs in preparing the President’s Fiscal Year 2004 budget. (They anticipate rating 
an additional 20 percent in each following year.) The PART includes general questions in 
each of four broad topics: 

(1) program purpose and design - whether it is focused, unique, and optimally 
designed; 

(2) strategic planning - whether long-term and annual goals are meaningful, 
coordinated with related programs, and aligned with the budget; 

(3) program management - i.e., financial and performance oversight and 
accountability; and 

(4) program results - whether the program has been evaluated, and is meeting its 
long-term and annual goals. 

OMB budget reviewers arrive at a profile for each program by reviewing information 
from budget submissions, agency strategic and annual performance plans, program 
evaluations, GAO and IG reports, and other sources, and make an overall assessment of 
whether a program is “effective”, “moderately” or “minimally” effective, or “ineffective”. 4 
OMB staff conducted assessments using the PART tool this September, with input from 
agency staff, in the first stage of preparing the President’s Budget for Fiscal Year 2004. 
The tool and instructions for using it are available at the same OMB website 
[ www.whitehouse.gov/omb/budintegration/part assessing2004.html 1. 

Integrating performance information into budget decisions is important because it can 
show the likely performance consequences of our budgetary choices. Our country faces 
long-term fiscal challenges related to spending on aging baby-boomers that will force 
increasingly difficult budgetary choices. We need to find ways to manage growth in 
spending on retirement and health programs, but we must also reexamine existing 
federal policies in light of the changing needs of the 21 st century. Many programs and 
policies were designed years ago to respond to earlier challenges and need to be 
scrutinized for their relevance to today. 



' Assessing Program Performance for the FY2004 Budget, Executive Office of the President, Office of 
Management and Budget, and OMB Memorandum M-02-10, Program Performance Assessments for the 
FY2004 Budget, dated July 16, 2002. 

http://www.whitehouse.gov/ombAudintegration/part_assessing2004.html 
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The federal budget process is a natural vehicle for discussing program performance 
because it provides an annual forum for review and debate of government priorities. As 
GAO testified this summer, “Performance-based budgeting can help enhance the 
government’s capacity to assess competing claims in the budget by arming budgetary 
decision makers with better information on the results of both individual programs as 
well as entire portfolios of tools and programs addressing common performance 
outcomes ”. 6 However, it’s important to temper expectations - both hopes and fears - for 
this process. Integrating program performance and results into budget discussions will 
not be a simple, straightforward undertaking. Performance information is only one input 
to budgeting. It cannot and will not replace political choices; it cannot settle debates 
about the appropriate role of the federal government or the need for various federal 
programs and policies. But performance information can inform the debate, focus the 
agenda on the outcomes we want to achieve, and clarify the likely performance 
consequences of resource decisions. In addition, performance problems don’t 
necessarily elicit a single budgetary response; they might prompt either budget cuts or 
enhanced investments and program redesign, depending on whether a poorly performing 
program is considered important to retain. 

In testimony this summer, GAO pointed out that the PART tool introduces transparency 
and opportunities for more in-depth discussion of program performance in OMB budget 
reviews, but also has some limitations . 6 Thorough reviews such as those implied by the 
PART tool obviously require credible performance information on outcomes. Decision 
makers must feel comfortable with the appropriateness of the goals and measures and 
accuracy of the data before they will use it to allocate resources. However, these have 
posed problems throughout implementation of GPRA. Some agencies struggle to 
measure meaningful outcomes and many can’t show their contribution to goals that are 
only partially influenced by federal programs. In addition, federal agencies are still 
working on developing reliable cost accounting data that is needed to link resources to 
results. 

Although the PART’S program-by-program approach fits with OMB’s agency-by-agency 
budget reviews, it is not well-suited to addressing cross-cutting issues or to looking at 
broad program areas in which several programs address a common goal. Yet it’s often 
critical to understand whether and how a program complements or supports other 
related programs. Obviously, it will be difficult to develop “bottom fine” judgments or 
ratings for programs with multiple performance goals and mixed performance records. 
At a minimum, reviewers should probably not focus on specific ‘scores’ on the PART 
instrument - particularly a “total” score summed across the diverse sections - but should 
use it as a profile of strengths and weaknesses for discussion. (How the ratings will be 
reported has been a continuing point of debate in public discussion of the tool.) Further, 
the achievement of federal or national policy goals often depends on the actions of state 
and local governments or non-governmental actors, which is noted but not fully captured 
in the PART. 



6 Performance Budgeting: Opportunities and Challenges (GAO-02-1 106T, Sept. 19, 2002). 
0 GAO-02-1 106T. Available at httD://www.eao.gov 
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On the other hand, the PART tool provides an opportunity for evaluations to capture 
more attention in OMB budget reviews and federal policymaking more generally. The 
tool promises to build on GPRA by using the performance data generated, and may 
increase the supply of data by promoting demand for it in the budget formulation 
process. An assessment tool won’t resolve policy debates, but it could help identify 
programs for further evaluation. The PART’S greatest contribution may be to focus 
discussions between OMB and the agencies on a given program’s progress towards 
planned performance, and what tools and strategies might be used to bring about 
improvements in achieving specific goals and objectives. For example, the PART section 
on management may identify ways to improve operations. The section on design might 
identify changes needed to improve targeting or effectiveness. The PART - and its 
sustained use over time - can also be expected to increase the attention paid to 
evaluation and performance information among federal agencies and the third parties 
involved in implementing federal initiatives. The fact that an agency’s score suffers from 
the absence of information may provide added impetus for agencies to enhance their 
evaluation and information-gathering capabilities. 

GAO’s September testimony also noted “the gaps and weaknesses [in performance and 
evaluation information] identified by the PART exercise may help pinpoint aspects of the 
federal evaluation infrastructure that need to be strengthened.” Experience with GPRA 
performance reporting has highlighted several challenges to assessing federal program 
outcomes that are already familiar to evaluators: 1) isolating a particular program’s 
contribution alongside other efforts to promote particular outcomes, 2) tracking 
outcomes that take years to achieve, and Unking them to annual budgets, 3) the 
diminished ability of federal agencies to influence outcomes of programs where 
responsibihty has devolved to the states, and 4) relying on states and other parties for 
the data with which to evaluate progress. Thus, the PART could be used to promote 
discussions of what is known and not known about program performance and stimulate 
evaluations especially designed to fill those gaps. Finally, GAO expressed concern that 
our reviews of federal agencies’ evaluation activities have found capacity to be quite 
uneven. GAO urged that investment in improving evaluation capacity be viewed as part 
of a broader initiative to improve the accountabihty and management capacity of federal 
agencies and programs. 

Moving forward, I’d like to suggest some ways that evaluators - within and outside the 
federal government - could assist agency and congressional decision makers in 
integrating evaluation and performance information into budget and policy discussions. I 
see four basic approaches: 1) Make your evaluations more relevant to this emerging 
federal budget process; 2) Help prepare agency staff for their program assessment 
discussions with OMB; 3) Develop tools to better inform the process as a whole; and 4) 
Improve the quality of state and local program data. 

(1) Make your evaluation more relevant. Situate your evaluation of a federal 
program in the context of the agency’s strategic and performance goals and 
expectations. Read the agency’s performance plans and discuss with project 
officers and program managers how they see their program contributing to their 
agency’s performance goals, and how your evaluation will fit into OMB’s 
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performance assessment. Which PART questions will your evaluation be 
expected to address? What other information will be available? What are their 
performance expectations for the program? What federal programs will this one 
be compared to and how is their performance measured? This may identify 
additional issues or measures to include in your evaluation or at least initiate a 
discussion of where else the agency might obtain data on those issues. 

(2) Help prepare agency staff for budget discussions. Work with your federal 
program managers to help them develop a comprehensive picture of where the 
program fits in the larger picture of efforts to achieve their outcome goals. Help 
them develop logic models that show the relationships between their program and 
agency annual and strategic goals. Articulate the distinctions between the 
contributions of their program and those of related programs. Model the 
relationship between their program and external factors that influence whether 
their goals are achieved. Help them to identify measures that more appropriately 
capture their program’s specific contribution to the relatively broad, general 
outcomes they often put forward. You probably can’t incorporate all these issues 
into your evaluation study, but you will have helped prepare agency staff to 
discuss these issues and explain how the resources spent on their program will 
contribute to achieving desired outcomes. 

- You could also help program managers identify gaps or weaknesses in their 
available performance information, identifying which key questions they have 
information on and which remain unanswered. Explore with them what targeted 
analysis might address those remaining questions. Again, this prepares them to 
engage in a more fruitful discussion of what crucial information they might obtain 
if they had additional funds for evaluation. 

(3) Develop tools to improve the budget and performance integration 
process. Contribute to a dialog on how to overcome key challenges to assessing 
federal program outcomes, such as: 1) isolating a program’s contribution to a 
broad effort composed of multiple programs and activities aimed at common 
goals; 2) assessing federal programs where decision authority has been devolved 
to the state and local levels; and 3) assessing the comparative advantage of 
current programs to alternative strategies for achieving a program’s goals. If the 
purpose of a flexible grant program is to help states and localities to meet their 
local needs, then should we be measuring the quality of those parties’ needs 
assessments and how well they target resources? Has the federal role evolved in 
these programs to providing intellectual leadership and technical assistance, to 
learn what works and help providers adopt successful practices? If so, then, 
should we evaluate how well agencies have achieved those goals, rather than 
measure progress in achieving national health, social or environmental outcomes? 

- We also need help in constructing an objective, balanced approach to conducting 
holistic assessments of programs, policies, and activities that are aimed at similar 
outcomes. OMB has launched an effort to develop common performance 
measures for programs in areas such as employment and training or food safety to 
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assist direct comparisons of their effectiveness. While there is a recognition that 
some programs may not in fact overlap because they serve distinct populations 
(e.g., general and special needs populations), I have not seen an approach for 
appropriately comparing programs that have different performance expectations 
for their different target populations. 

(4) Improve the quality of state and local data. For those of you who do not 
work directly with federal programs or agencies, we could still use your help in 
improving the quality of state and local program data. You can help program staff 
select appropriate measures of their goals and objectives or adopt data collection, 
processing, and reporting procedures to help ensure their data validity and 
reliability. Encourage them to collaborate with program partners and colleagues 
in other locales in selecting measures, data collection technologies and 
definitions. Elaine Vaurio and I saw an excellent example of coordination in the 
USDA’s Expanded Food and Nutrition Education Program. 7 State and local 
program providers banded together, with federal coordination and leadership, to 
draw on research literature and experts to develop common measures of client 
benefits that were sophisticated, meaningful and useful at the local, state, and 
federal levels. 

Periodic re-examination of federal government priorities, programs and activities is an 
important responsibility of the executive and congressional branches - to ensure 
accountability to the public and to meet the nation’s needs. However, effective oversight 
is difficult and requires both good quality information and a sophisticated understanding 
of how programs work. I believe evaluators have important roles to play in helping to 
meet those needs. 



? This example was described in another paper on this panel and in the GAO report, Program Evaluation: 
Strategies for Assessing How Information Dissemination Contributes to Agency Goals (GAO-02-923, Sept. 
30, 2002). Available at http://www. gao. gov 
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